blobstore: support parallel download option and createParentPath option in AWS and GCP by LihaoLiuXs · Pull Request #377 · salesforce/multicloudj

LihaoLiuXs · 2026-04-08T18:27:34Z

Summary

Moved the parallelDownd option to asyncClient. Since we don't have the conformance test for async client. I tested it locally with example.

GCP test result:
BUCKET_NAME=palsfdc mvn -q exec:java -Dexec.mainClass=com.salesforce.multicloudj.blob.Main -Dbucket.name=palsfdc 2>&1
=== STARTING PARALLEL DOWNLOAD TESTS ===
Provider: gcp
=== Testing Parallel Download (Path) ===
SLF4J(W): No SLF4J providers were found.
SLF4J(W): Defaulting to no-operation (NOP) logger implementation
SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
SLF4J(W): Class path contains SLF4J bindings targeting slf4j-api versions 1.7.x or earlier.
SLF4J(W): Ignoring binding found at [jar:file:/Users/barry.liu/.m2/repository/org/slf4j/slf4j-simple/1.7.2/slf4j-simple-1.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J(W): See https://www.slf4j.org/codes.html#ignoredBindings for an explanation.
[parallelDownloadToPath] Uploading test blob: key=parallel-download-path-1776870394493.bin, size=16777216
[parallelDownloadToPath] key=parallel-download-path-1776870394493.bin, bytes=16777216, elapsedMs=1721, etag=CNLan4begZQDEAE=
[parallelDownloadToPath] SUCCESS
=== Testing Parallel Download (createParentPath) ===
[parallelDownloadWithCreateParentPath] Uploading nested-key blob: key=parallel/nested/dir/blob-1776870398374.bin
[parallelDownloadWithCreateParentPath] resolvedPath=/tmp/parallel-download-root-1776870398374/parallel/nested/dir/blob-1776870398374.bin, bytes=8388608, etag=COLJzofegZQDEAE=
[parallelDownloadWithCreateParentPath] SUCCESS
=== Testing Parallel Download (with Range) ===
[parallelDownloadWithRange] key=parallel-download-range-1776870400563.bin, bytes=3073, expected=3073
[parallelDownloadWithRange] SUCCESS
=== Testing Parallel vs Sequential Download ===
[parallelVsSequentialDownload] sequentialMs=2349, parallelMs=2422, sizeBytes=33554432, seqDownloadedBytes=33554432, parDownloadedBytes=33554432
[parallelVsSequentialDownload] SUCCESS
Parallel download tests completed!

AWS test result:

AWS_ACCESS_KEY_ID=ASIAZQ3DQ6RHW2RLTYBV AWS_SECRET_ACCESS_KEY=ptpSaDk3289DvAAa1a2nWydKWorUHOCv6C5EZirZ AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjEIj//////////wEaCXVzLWVhc3QtMSJHMEUCIQDkYobo2adHLDVDa+nGaRy4fMqPI+Kn5JXDOQuNqtJbCwIgGG0YDYe4mssxmxR0SXi05yynyLm9iMOoqCIgmBgBYz8qoQMIURAAGgw2NTQ2NTQzNzA4OTUiDDehr5JzqAAf6rhQNir+AhcU5dA7Rvp02CwPTI5f2u6StBC4zzKXwhmfHSFvhDr94unO6627FuL2s0TZTqoTWzCiMPeUlDbQsAECdnbMx+2WUyNNcMx3arePqqaH2WBTY8CnSTLEYIZ/JKUrvs9oSCC8gw79hW37tlQVN6fJWlHu4LnXQcECnt6NUCGH0TC4P3udxZs52GZ7cNxyxhhT8XULEuaz2iGPmDbVNW/aLT8ACAxZqnuYS0P0s/Fu7RWdOI2JW2Yjavt6+rq/ndDMzZpBKXTfGxCV0i9XPIpLhRA2VXAI9gZ5vqNSXVPpwhSFRViATPvppkHFOZ0j2K0ZnqOzo0ZgwTzcbPemo1e1retBNX0p5o7q6CIpwdcF549NmVvAGmvrPRQuYWEJUnMKRvTkNgPR0B6qCT4Nnr0zm4JJwXYpgPtns9gzfIngj/mjFxIK2reP92G/YKh/DDGeFt21sK93SyZmDk50qu+58TKwrNcMG88O9+hkltvhySc+o8b8Gu5cpbyCTZiggLIwy92jzwY63gFbDq00OpQM0b4/EsIVRKQLF6ONqU+QYqaRX1I9n/4TgKLz1/vYVOwb34450QreE8PakIhIQlnfPv2JNlQh8SzLmOGSF8BHX1+3rh+pEU9rPTo3mjJXts18LMtnFwP/ExOJ/A9dgWdQzRgX6GN2akSgC5pdnpoKZaGMVm5Ft7fU96EGl3D1cUiwcmrGHBHCbGeeND54tMoA+Xx/XhO7uZjdwlHHjMKgJuZmmOp76Nd/IFdBZ7Gfw9kfXfZdinVS8ilxzOJFzhvM0/jBIEL1fDFxmUgXKSUEaa0pEAiXOqg= AWS_REGION=us-west-2 BUCKET_NAME=example-test-bucket-barry BUCKET_REGION=us-west-2 mvn -q exec:java -Dexec.mainClass=com.salesforce.multicloudj.blob.Main -Dprovider=aws -Dbucket.name=example-test-bucket-barry -Dbucket.region=us-west-2 2>&1 | grep -vE '^\s+at |Caused by:' | tail -60
=== STARTING PARALLEL DOWNLOAD TESTS ===
Provider: aws
=== Testing Parallel Download (Path) ===
SLF4J(W): No SLF4J providers were found.
SLF4J(W): Defaulting to no-operation (NOP) logger implementation
SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
SLF4J(W): Class path contains SLF4J bindings targeting slf4j-api versions 1.7.x or earlier.
SLF4J(W): Ignoring binding found at [jar:file:/Users/barry.liu/.m2/repository/org/slf4j/slf4j-simple/1.7.2/slf4j-simple-1.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J(W): See https://www.slf4j.org/codes.html#ignoredBindings for an explanation.
[parallelDownloadToPath] Uploading test blob: key=parallel-download-path-1776873590056.bin, size=16777216
[parallelDownloadToPath] key=parallel-download-path-1776873590056.bin, bytes=16777216, elapsedMs=2067, etag="e06888832b83d4f381e885edebd1d782"
[parallelDownloadToPath] SUCCESS
[parallelDownloadToPath] remote blob retained for inspection: parallel-download-path-1776873590056.bin
=== Testing Parallel Download (createParentPath) ===
[parallelDownloadWithCreateParentPath] Uploading nested-key blob: key=parallel/nested/dir/blob-1776873595717.bin
[parallelDownloadWithCreateParentPath] resolvedPath=/tmp/parallel-download-root-1776873595717/parallel/nested/dir/blob-1776873595717.bin, bytes=8388608, etag="57b019a28c426df5727b3992701bd2be"
[parallelDownloadWithCreateParentPath] SUCCESS
[parallelDownloadWithCreateParentPath] remote blob retained for inspection: parallel/nested/dir/blob-1776873595717.bin
=== Testing Parallel Download (with Range) ===
[parallelDownloadWithRange] key=parallel-download-range-1776873598786.bin, bytes=3073, expected=3073
[parallelDownloadWithRange] SUCCESS
[parallelDownloadWithRange] remote blob retained for inspection: parallel-download-range-1776873598786.bin
=== Testing Parallel vs Sequential Download ===
[parallelVsSequentialDownload] sequentialMs=3321, parallelMs=3580, sizeBytes=33554432, seqDownloadedBytes=33554432, parDownloadedBytes=33554432
[parallelVsSequentialDownload] SUCCESS
[parallelVsSequentialDownload] remote blob retained for inspection: parallel-vs-sequential-1776873600509.bin
Parallel download tests completed!

Some conventions to follow

add the module name as a prefix
- for example: add a prefix: docstore: for document store module, blobstore for Blob Store module
for a test only PR, add test:
for a perf improvement only PR, add perf:
for a refactoring only PR, add "refactor:"

codecov-commenter · 2026-04-08T20:40:11Z

Codecov Report

❌ Patch coverage is 37.66234% with 96 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.49%. Comparing base (d8556bc) to head (26ce906).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
.../salesforce/multicloudj/blob/gcp/GcpBlobStore.java	26.80%	62 Missing and 9 partials ⚠️
...oudj/blob/async/driver/AbstractAsyncBlobStore.java	10.00%	8 Missing and 1 partial ⚠️
.../multicloudj/blob/aws/async/AwsAsyncBlobStore.java	33.33%	5 Missing and 1 partial ⚠️
...alesforce/multicloudj/blob/aws/AwsTransformer.java	0.00%	4 Missing ⚠️
...rce/multicloudj/blob/driver/AbstractBlobStore.java	70.00%	2 Missing and 1 partial ⚠️
...e/multicloudj/blob/inmemory/InMemoryBlobStore.java	75.00%	2 Missing and 1 partial ⚠️

❌ Your patch status has failed because the patch coverage (37.66%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #377      +/-   ##
============================================
- Coverage     82.10%   81.49%   -0.62%     
- Complexity      641      643       +2     
============================================
  Files           194      194              
  Lines         11798    11994     +196     
  Branches       1572     1605      +33     
============================================
+ Hits           9687     9774      +87     
- Misses         1434     1530      +96     
- Partials        677      690      +13

Flag	Coverage Δ
unittests	`81.49% <37.66%> (-0.62%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

iamabhilaksh · 2026-04-09T10:59:44Z

This looks solid. Could you share the testing logs or a brief summary of how this was verified on AWS/GCP? Especially interested in seeing the parallel download and createParentPath behavior in action. 👍

sandeepvinayak · 2026-04-21T06:23:20Z

 import software.amazon.awssdk.http.apache.ApacheHttpClient;
 import software.amazon.awssdk.http.apache.ProxyConfiguration;
 import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.s3.S3AsyncClient;


that's a bit odd - we shouldn't mix sync and async in here. AwsBlobStore in meant for sync operations

move the parallel download option to asyncClient.

sandeepvinayak · 2026-04-21T06:24:07Z

+import software.amazon.awssdk.transfer.s3.S3TransferManager;
+import software.amazon.awssdk.transfer.s3.model.DownloadFileRequest;


sandeepvinayak

if the client wants to do parallel downloads - they need to use async client. We shouldn't mix these two - it will get into the high maintainance sphagetti code.

LihaoLiuXs · 2026-04-22T19:35:23Z

if the client wants to do parallel downloads - they need to use async client. We shouldn't mix these two - it will get into the high maintainance sphagetti code.

Agreed. I moved the parallel download option to asyncClient.

sandeepvinayak · 2026-04-24T19:25:33Z

+   * When createParentPath is enabled, resolves the final file path by appending
+   * the object key to the destination root and creating any missing parent directories.
+   */
+  private Path resolveDownloadDestinationPath(DownloadRequest request, Path destination) {


it;s not resolving, it's creating.

Updated the name.

sandeepvinayak · 2026-04-24T19:26:57Z

+    Path destinationPath = resolveDownloadDestinationPath(request, path);
+    GetObjectRequest getObjectRequest = transformer.toRequest(request);
+    if (request.isParallelDownload()) {
+      DownloadFileRequest downloadFileRequest =


we should move it to transformer

Moved this to transformer.

sandeepvinayak · 2026-04-24T19:27:49Z

+    }
+
+    /**
+     * (Optional) If true, provider implementations may preserve the key's parent path structure in


i think it creates paths and we should explicitly mention it

Update the comment for this

sandeepvinayak · 2026-04-24T19:28:42Z


+  /**
+   * Worker count for {@link TransferManager} divide-and-conquer downloads, analogous to tuning
+   * gsutil sliced / parallel Range GET downloads for large objects (see Google Cloud Performance


let's remove any unlrelated details,we have nothing to do gsutil

Remove the unrelated comments

sandeepvinayak · 2026-04-24T19:29:03Z

+   * gsutil sliced / parallel Range GET downloads for large objects (see Google Cloud Performance
+   * Atlas: Optimizing your Cloud Storage performance).
+   */
+  private static final int PARALLEL_LARGE_FILE_DOWNLOAD_MAX_WORKERS = 8;


how do we chose 8 ?

8 workers will give us 8 workers × 16 MiB chunks ≈ 800 MB/s aggregate ≈ 6.4 Gbps, which saturates the GCE Compute Engine default per-VM egress cap (7 Gbps).

For machine series that support multiple physical NICs, such as A3, A4, A4X, and A4X Max instances, 7 Gbps per NIC: https://docs.cloud.google.com/compute/docs/network-bandwidth#egress_outside_vpc

Number of work options: 4, 8, 16, 22, 32, 48. transfer_manager sliced download is slow on cloud run googleapis/python-storage#1093

sandeepvinayak · 2026-04-24T19:29:33Z

+   */
+  private static final int PARALLEL_LARGE_FILE_DOWNLOAD_MAX_WORKERS = 8;
+
+  private static final String PARALLEL_STRIP_PREFIX_TO_BASENAME = "^.*/";


the var name and value doesn't align

This value matches everything up to and including the last '/' in an object key. For example for object key data/2026/04/report.csv, "^.*/" will match the data/2026/04/, and as result we will only keep the report.csv. I will rename this var to OBJECT_KEY_DIRECTORY_PREFIX_REGEX.

sandeepvinayak · 2026-04-24T19:30:09Z

+  // parallelDownload not supported: delegates to OutputStream which
+  // cannot accept parallel range-GET writes.


lets remove this comment, the docs are added in abstract

sandeepvinayak · 2026-04-24T19:31:16Z

+  private Path resolveDownloadDestinationPath(DownloadRequest request, Path destination) {
+    if (!request.isCreateParentPath()) {
+      return destination;
+    }
+    // Keep key parent structure under the provided local destination root when requested.
+    Path resolved = destination.resolve(request.getKey()).normalize();
+    Path parent = resolved.getParent();
+    if (parent != null) {
+      try {
+        Files.createDirectories(parent);
+      } catch (IOException e) {
+        throw new SubstrateSdkException("Failed to create destination directories", e);
+      }
+    }
+    return resolved;
+  }


since this is already common, should go to abstract class

Moved this to AbstractBlobStore and AbstractAsyncBlobStore.

LihaoLiuXs added 5 commits April 1, 2026 12:30

Blobstore: support parallel download option

f695b88

Blobstore: support parallel download option and createParentPath option

2000068

Implement the option for parallelDownload and createParentPath

d9efd05

Update the aws tests

8c33f3d

Add conformace test record files for aws and gcp

0af89b9

salesforce-cla Bot added the cla:signed label Apr 8, 2026

LihaoLiuXs and others added 3 commits April 8, 2026 14:28

Merge branch 'main' into parallelDownload

0dbf04e

Fix the test failure

6d8b86f

Fix the conformance test in gcp

2d7a990

LihaoLiuXs and others added 3 commits April 9, 2026 15:28

Merge branch 'main' into parallelDownload

87261f4

code reformat

3264815

Fix the checksytle

b100526

sandeepvinayak reviewed Apr 21, 2026

View reviewed changes

sandeepvinayak requested changes Apr 21, 2026

View reviewed changes

LihaoLiuXs and others added 4 commits April 21, 2026 23:38

move the parallel download option to asyncClient

24a8a49

Remove the conformance test recrod files

3b7558a

Merge branch 'main' into parallelDownload

ae2fb7f

Fix checkstyle

12e9e31

sandeepvinayak reviewed Apr 24, 2026

View reviewed changes

LihaoLiuXs added 2 commits April 24, 2026 17:42

Remove the getAsyncProxyClientBuilder which is not needed

7394d70

Address the comments

26ce906

		import software.amazon.awssdk.transfer.s3.S3TransferManager;
		import software.amazon.awssdk.transfer.s3.model.DownloadFileRequest;

		// parallelDownload not supported: delegates to OutputStream which
		// cannot accept parallel range-GET writes.

Conversation

LihaoLiuXs commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Some conventions to follow

Uh oh!

codecov-commenter commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

iamabhilaksh commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LihaoLiuXs Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sandeepvinayak left a comment

Choose a reason for hiding this comment

Uh oh!

LihaoLiuXs commented Apr 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

LihaoLiuXs commented Apr 8, 2026 •

edited

Loading

codecov-commenter commented Apr 8, 2026 •

edited

Loading

iamabhilaksh commented Apr 9, 2026 •

edited

Loading

LihaoLiuXs Apr 22, 2026 •

edited

Loading